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This study presents a meta-evaluation of a beta-test of a customer serx’ice training program. The initial 
evaluation showed a low response rate. Therefore, the meta-evaluation focused on issues related to the 
conduct of the initial evaluation and reasons for nonresponse. The meta-evaluation identified solutions to 
the nonresponse problem as related to authority, capacity, and motivation. 
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Organizations spend several billion dollars each year purchasing outside training materials and services (Galvin, 
2001). In turn, organizations that specialize in producing these training materials often spend thousands of dollars to 
develop, test, and market such programs. Although creativity in concept and design may be critical during the 
development stage, testing the product’s concept and the product’s usability and effectiveness are essential for 
ensuring a marketable product. For example, Dolan and Matthews (1993) write that product testing can help 
determine the quality of the product’s concept, reduce or eliminate problems prior to market introduction, and 
provide one part of a sales promotion effort. Therefore, an essential component of testing new products, such as a 
new training program, is obtaining evaluative feedback on various aspects of the program. 

Our purpose is to detail the experience of one organization that attempted to implement several beta-tests of a 
newly designed training program. We begin by describing different forms of product testing and the problems 
associated with obtaining a good response rate to the data collection instruments. We then outline the evaluation 
procedures used to implement a beta-test of the new training program. Because of problems with nonresponse to the 
beta-tests, we then present a meta-evaluation that was undertaken of the beta-tests and the resulting description of 
the successes and failures in obtaining needed participation. We end the article with reflections on what happened in 
the initial beta-tests, offer suggestions for improving evaluative feedback for future beta-testing and evaluation 
endeavors, and discuss the need for future meta-evaluations to improve organizational learning. 

Product Testing 

Three types of product testing are typical in many product development efforts. These are: 1) usability testing, 
2) pilot testing, and 3) beta-testing. Usability testing and usability labs (e.g., Barnum & Dragga, 2001, Coe, 1996, 
Rubin, 1994) have received increased attention in recent years. In most cases, usability testing has been used in 
testing computer-based products (Dumas & Redish, 1999) with the goal of making the product more user friendly. 

Pilot-tests and beta-tests are concerned with more than the usability of the product. These tests tend to focus 
more on the outcomes or impact from using the product. Thus, with a training product, a usability test would 
determine whether the instructor could effectively use the training materials and whether the activities were well 
designed. A pilot- or beta-test however, would also examine the instructors’ and trainees’ reactions to the training 
and quite possibly, the extent to which trainees have learned from their participation in the program, as well as the 
ways in which they have been able to apply their new knowledge and skills back on the job. 

Although the literature uses the terms “pilot-test” and “beta-test” interchangeably (e.g., Chernick, 1992; Howe 
& Perlstein, 1995; Rosenkrans, 2001, Wurangian, 1993), subtle differences exist between the two activities. Dolan 
and Matthews (1993) point to one important distinction: Product design should be frozen prior to the beta-testing 
phase. This means that pilot testing is designed to try out and accept or reject new ideas and procedures. Once the 
product is in near final or final form, a beta-testing phase can begin. Thus, beta-testing means testing the final 
product in real-life situations. Even more than a usability test or a pilot-test, beta testing in the for-profit world can 
serve as part of the sales promotion effort. The beta-test allows customers to use the final product and provides the 
supplier organization an opportunity to obtain data or testimonials to spur future product sales. 

With such benefits accruing from usability testing, pilot- and beta-testing, one might imagine that many such 
tests occur. This is often not the case, however. In order to obtain quality beta-test feedback, participants must be 
willing and able to respond to data collection activities. Obtaining a high response rate to such requests can be a 
significant challenge. 
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In the next section we describe the problems of non-response by organizations and individuals to these various 
forms of testing and place these problems within the survey non-response literature. Non-response to Pilot-Testing 
and Beta-Testing Undoubtedly, pilot testing and beta testing are most successful with organizations, groups, and 
individuals that reflect the target market for the product. However, obtaining a representative sample is not always 
easy. This is particularly true in testing training programs for at least three reasons: (1) Decision-makers within the 
organization must be convinced that the training program will serve the organization — for example, the goals and 
objectives of the training should help fulfill a particular strategic or business objective. Or, in Swanson's (2001) 
words, it must “be connected to the fundamental mission and outputs of the organization” (p. 2); (2) Decision- 
makers, and others within the training function and potentially throughout the organization, must be convinced that 
the new program will yield positive results for the trainees and the organization; (3) Then, having persuaded 
decision-makers and those in the training function of the value of participating in the pilot- or beta-test, the trainees 
must be convinced that it will be worth their time and effort. Once an organization has agreed to participate in a 
pilot- or beta-test, the next challenge is obtaining evaluative feedback once the program has been delivered. Most 
beta-tests involve the administration of surveys that ask participants to reflect on their experience with the product or 
program. While these may be distributed face-to-face during or at the end of the testing period, surveys are often 
used to follow-up with participants days, weeks, or months later. 

The difficulty faced by beta-test administrators in getting organizations and individuals to participate and 
respond to beta-test evaluation surveys, fits within the theoretical framework encompassed by the organizational 
survey nonresponse literature. As discussed by Tomaskovic-Devey, Leiter, and Thompson (1994), organizational 
surveys often have somewhat low response rates. Indeed, a response rate in the 30 percent range is fairly common 
(e.g., Blau, Falbe, McKinley, & Tracy, 1977; Lincoln & Kalleberg 1985; Russ-Eft, 1997) 

Tomaskovic-Dewey, et al. (1994) observe that since organizations are hierarchical, possess differentiated 
functional groups, and embody routine patterns of behavior, a member’s willingness to participate in an 
organizational survey (or in this case, a pilot- or beta-test) depends on three factors: (1) their authority to respond, 
(2) their capacity to respond, and their (3) motivation to respond. Furthermore, these researchers hypothesize that all 
three conditions must be present for individuals within an organization to willingly respond to a survey request. For 
example, the CEO of an organization may have the authority and the motivation to respond but, given the various 
functional groups within the organization, may not have the needed information to respond. Another example, in 
beta-testing a training product, the VP of Human Resources may have the authority and the motivation to request 
such a test, but he or she may not have the ability to orchestrate the various departmental and functional heads to 
agree to participate. Furthermore, even if the VP were able to obtain the cooperation from these departmental and 
functional heads, the individual trainees may lack the motivation needed to complete the best-test training surveys. 
Evaluating Beta-test Training Programs at HIJ Training Sendees 

As part of the product development process, HIJ Training Services (a pseudonym rather than the actual name of the 
company) undertakes extensive pilot testing as well as beta testing of its new training programs in order to 
determine their effectiveness and marketability. Program development staff usually conducts pilot-tests first with in- 
house staff, and then with volunteer client organizations. Both during and after the pilot-test, problems, or areas of 
needed revision are noted. Changes are then made to the design, materials, and guidelines for implementation. Some 
programs have been subjected to at least five pilot-tests before a beta-test has been conducted. At HIJ Training 
Services, a program is ready for beta testing when the product is considered “stable” and is nearly out the door. The 
goals of the beta-test are to determine the extent to which the training program’s objectives are met and the target 
behaviors are affected. A training program’s beta-test typically involves inviting clients to participate in the training, 
and then asking them to complete a series of evaluation surveys, mastery tests, and phone interviews. In return for 
their participation in the beta-test evaluation, the participating organizations either receive product credit, or in the 
case of this study, free training. 

The focus of this study was a newly designed customer service program that replaced two older customer 
service programs the organization used to offer. Since the new training program focuses on behaviors exhibited by 
customer service representatives, HIJ Services was interested in obtaining data on behavioral changes that resulted 
from the training. Thus, one objective of the beta-test evaluation was to obtain evidence regarding the extent of 
participants’ behavioral change as a direct result of their training experience. The goals of the new customer service 
training program are to: Build a service-oriented culture within an organization, provide basic skills for entry-level 
service providers, so that they can effectively manage routine interactions with customers or clients, and provide 
advanced skills for more experienced service providers, so that they can improvise, when needed, in order to meet a 
wider range of customer or client needs. 

To recruit beta-test sites, HIJ Training Services salespeople were sent an email asking them to nominate 
organizations that might be interested in participating in the training. Eight salespeople responded saying they knew 
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of an organization that would be a good candidate for the training program. The salespeople then provided the 
Evaluation Department with the name of a “Client Contact” who was responsible for working with HIJ Training 
Services staff to schedule the training and recruit trainees. In return for the free training, the training participants and 
their managers were asked to participate in the following beta-test evaluation activities: 

Training Participants: 

At the beginning of the training a survey (referred to as a pretest) asking them to self-assess their pretraining 
behaviors. During the training, and immediately after each module a posttest to assess their mastery of the 
material, at the end of training, a post course reaction survey was administered. Approximately 3-4 weeks after 
training sent via the mail a post training survey to self assess their behaviors relative to the program content. 
Additional mastery tests for each of the modules they completed were administered. Approximately 3-4 weeks 
post training was conducted. Telephone interviews to determine if the participants had been able to use what 
they learned on the job (referred to as an ROI interview) 

Participants’ Managers: 

• Prior to the training: A survey asking them to assessing the participants' pretraining service behaviors 

• Approximately 3-4 weeks after training sent via the mail: Another survey asking them to assess the 
participants service behaviors 

• Approximately 3-4 weeks post training: Telephone interviews to determine if the participants had been able 
to use what they learned on the job (referred to as an ROI interview) 

It was anticipated that each of the eight organizations would fully honor its commitment to participate in all 
beta-test evaluation activities since they knew they were receiving the training at no cost. However, the response rate 
for the post-training evaluation activities was far less than expected. For example, the response rates for the trainees’ 
post-training survey (three to four weeks after training) were: 0%, 0%, 18%, 43%, 56%, 69%, 82%, and 83%. For 
the managers it was: 0%, 0%, 0%, 44%, 71%, 86%, 120%, and 150% (note: some managers completed the post test, 
but not the pretest, thus explaining the 120% and 150% figures). A review of the response rates indicates that in 
organizations where participants completed and returned the evaluation instruments so did their managers. While 
concerted efforts were made to conduct interviews with the managers three to four weeks after the training, only 
three managers were contacted and interviewed. The rest did not respond to repeated telephone and email messages. 

Given that the beta testing of new programs is considered instrumental to HIJ Training Services’ product design 
and marketing activities, the Evaluation Department wanted to understand why some of the beta-test organizations 
did not fully comply with the requirements to evaluate, as was a condition of their participation in the training. They 
also hoped to learn ways in which they might increase the response rate of future beta-test evaluation efforts. Such 
questions led to the undertaking of a meta-evaluation or an evaluation of the evaluation (Stufflebeam, 1979; Scriven, 
1991). 

Research Questions 

Most meta-evaluations (e.g., Curran, Christopher, Lemire, Collins, & Barrett, 2003; Grasso, 1999) tend to use The 
Program Evaluation Standards (Joint Committee on Standards for Educational Evaluations, 1994) or the “Guiding 
Principles for Evaluators” (Shadish, Newman, Scheirer, & Wye, 1995). Rather than use some pre-established 
framework, however, the present meta-evaluation focused specifically on the stakeholder questions (Preskill & 
Russ-Eft, 2005; Russ-Eft & Preskill, 2001). By doing so, the meta-evaluation results would, hopefully, lead to 
increased use and organizational learning (Preskill & Torres, 1999). Thus, the following questions were explored: 

1). How were clients invited to participate in the training and its evaluation? 2). Why did clients agree to participate 
in the beta-test training evaluation? 3). How did the client contacts work with the facilitators in explaining the 
evaluation's requirements? 4). Why did some clients not fulfill their commitment to complete all of the evaluation's 
requirements? 5). What kind of incentives would have made clients more likely to fully participate in the 
evaluation? 6). For those clients who participated more fully in the evaluation process, what factors contributed to 
their participation? 

Research Design and Data Collection Methods 

This study was conducted using a qualitative case study approach. Data were collected via telephone interviews and 
a review of archival data. 

Interviews 

Ten - 45-minute telephone interviews were conducted with 24 individuals: Six of the eight HIJ Training Services 
salespeople who invited client organizations to participate in the beta-test training. (Two salespeople did not 
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participate — one had left the company with no forwarding contact information, and one said she was too busy to 
participate). Six of the eight Client Contacts within the participating organizations (Two Client Contacts did not 
participate — one had retired from the organization, and in one case the salesperson asked that the client not be 
called. This salesperson chose not to participate in this study.) 

Five of the seven Facilitators (trainers) who delivered the beta-test program (One trainer delivered the program 
in two organizations, one trainer could not be located, and the other was unreachable.) Three Managers from one 
participating organization, the Customer Service Training program designer, the external consultant who developed 
the program’s evaluation instruments, and two staff members from the H1J Training Services’ Evaluation 
Department. 

Archival Data 

The following records, documents, and materials were also reviewed: The Customer Service Training evaluation 
instruments and cover letters, the Customer Service Training program materials, e-mail communications between 
F11J Training Services and the eight client organizations, and evaluation instrument completion records 

An Evaluation Department staff member sent the eight salespeople an email message that described the 
purpose of the research study, and explained that the researcher would be calling them to schedule a telephone 
interview. To begin, one of the researchers (second author) interviewed the program’s designer and evaluation staff. 
From there, each salesperson was contacted and interviewed. Once all of these interviews were completed, the 
Client Contacts and Facilitators were interviewed. Conducting the interviews in this order provided the support and 
context for each of the subsequent interviews, which in some cases was important for interpreting each person's 
comments. Given everyone’s busy schedules, some individuals were called four to six times before an interview was 
finally completed. 

Findings 

Research Question #1: How were clients invited to participate in the training and its evaluation? 

The salespeople were primarily motivated to involve client organizations in the beta-test because: (a) they thought it 
would be a good fit for the client, (b) they wanted to establish a relationship with a new client, or (c) they thought 
the client’s name and reputation would enhance the marketability of the program. For example, one person said. “I 
wanted to establish a credible client relationship and thought the program would fit their needs.” The salespeople’s 
motivation to select certain organizations appeared to stem from their interest in developing a new client relationship 
or in supporting an existing relationship rather than in supporting the evaluation of the beta-test training. This likely 
contributed to the varying levels of commitment from the client organizations. 

Research Question #2: Why did clients agree to participate in the beta-test training evaluation? 

The beta-test organizations saw their participation in the study as an opportunity to experience the training without 
making any financial investment. One Client Contact explained, “We were looking for a customer service program. 
We sent key people to the pilot and it gave us an opportunity to get a good hard look at it.” Each organization also 
reported having a need for customer service training at that time. One Client Contact said, “There was a huge push 
for improving customer service. It was offered up to us and there was quite a bit of commitment... they pulled people 
in from all over.” Depending on the level of communication and contact with the salespeople, the Client Contacts’ 
understanding of their role with regard to the evaluation varied significantly. In some cases, the original Client 
Contact either left the organization, or handed off the evaluation's responsibility to another employee who was ill 
informed about HIJ Training Services’ expectations for the evaluation. 

Research Question #3: How did the Client Contacts work with the facilitators in explaining the evaluation’s 
requirements? 

In four of the organizations. Client Contacts reported that they were told little if anything about what they 
would be expected to do regarding the program’s evaluation. For example, in an organization that had a very low 
response rate, the Client Contact explained that while she was aware of the evaluation requirement, she handed the 
evaluation surveys off to another office for administration. She admitted that the person who was asked to disttibute 
and collect the surveys probably didn’t realize the importance of the surveys or the agreement the organization had 
made with HU Services. 

As a result, some Client Contacts did not fully understand the organization's commitment to participate in the 
evaluation as a condition of their receiving the free training. Moreover, these individuals were not particularly clear 
about their role in the evaluation process. Given this situation, it is likely that the trainees were also unclear about 
their expectations for completing the evaluation instruments. 

Research Question #4: Why did some clients not fulfill their commitment to complete all of the evaluation’s 
requirements? 
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Training program participants were asked to complete several post-training evaluation surveys that arrived at 
the same time. Their managers were asked to do the same. The following reasons were provided in an attempt to 
explain why the trainees and their managers failed to respond to the post training surveys and manager interviews: 
They were too busy, they left the organization, they changed jobs within the organization, office was shut down, 
lack of manager follow-through and commitment to the training and/or the evaluation, lack of Client Contact 
involvement, lack of participants’ understanding about H1J Training Service’s expectations for feedback, and lack of 
motivation; reason to provide feedback 

One Client Contact explained, “There is a cultural arrogance in the organization. They perceive themselves to be 
very, very busy all of the time. And. it wouldn't fit on their radar screen.” Another offered the following 
explanation, “This organization is a topsy-turvy place. Management’s style is such that no one has to do anything; 
participation is voluntary for everything.” 

It may also be possible that customer service employees change jobs more frequently than employees in other 
fields, or that the demands of their jobs affect their ability or motivation to respond to evaluation requests. In 
addition, some interviewees think it’s possible that the number of evaluation instruments affected the lower than 
expected return rate. 

Research Question #5: What kind of incentives would have made clients more likely to fully participate in the 
evaluation ? 

Some salespeople. Client Contacts, and Managers believe that individual incentives should not be necessary if the 
organization commits to participating in the beta-test training program and its evaluation, and clearly communicates 
this to the trainees and their managers. As one salesperson said, “It’s a real bonus to get the training free; they don’t 
need anything else.” A Manager added, “If people are told to do something, they should do it.” At the same time, 
several interviewees suggested that individual incentives would motivate trainees in particular, to complete the post 
training evaluation surveys. One Client Contact suggested, “Individual incentives are a good idea - something like a 
gift certificate to amazon.com.” An Salesperson agreed, saying, “Personal incentives, like gift certificates would be 
good. It wasn't enough that we were going to give them free training, because it didn’t mean anything to them.” 
Some of the incentives mentioned, besides gift certificates, include: money, office items with HII Training Service’s 
logo, free training or consulting, books, and product certifications. In addition to incentives, the Salespersons and 
Client Contacts suggested that sending trainees and their managers' frequent emails reminding them to respond to 
the evaluation requests would have been a good idea. 

Research Question #6: For those clients who participated more fully in the evaluation process, what factors 
contributed to their participation? 

In the three organizations that had high survey response rates, it appears that in each case, there was a good 
relationship between the HIJ Training Services salespeople and the client organization. In two of the cases, these 
relationships had been ongoing for a number of years. Maintaining a good working relationship seemed important to 
the Client Contacts. As one Client Contact explained, “We've had a good relationship with HIJ Training Services 
for along time, and didn’t want to hurt the relationship. Helps them and us. Tried to do our best following through.” 
Relatedly, the client organizations' experience with HU Training Service’s products might have also influenced 
these organizations to participate more fully. And, in cases where the Client Contact made a concerted effort to 
position the training and its evaluation with participants and their managers, higher response rates were achieved. 

To summarize, the organizations that participated more fully in the beta-test evaluation activities had the 
following characteristics: 

• Client contacts had prior positive experience with HIJ Training Service’s products and services. 

• Organizations felt privileged to be a beta-test site. 

• The climate within the organization was stable during the beta-test; that is, the organization did not 
experience any major disruption due to merger, office closure, or layoffs. 

• Salespeople understood and clearly communicated the organization’s role in the evaluation. 

• Client Contacts believed it was the organization’s obligation to fully comply with the evaluation. 

• Client Contacts were clear about what was required for the evaluation and remained in their position 
throughout the training program’s delivery and evaluation phases. 

• Client Contacts reminded trainees and their managers (often several times) to complete the evaluation 
forms, and how important their feedback was to HIJ Training Services. 

• Client Contacts and Managers believed that it was enough of an incentive to receive the free training — that 
no individual incentives for completing and returning the evaluation forms were necessary. 

• Facilitators/Trainers developed a good relationship with the trainees. 
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• Facilitators/Trainers emphasized throughout the training program that it was a beta-test, and how important 
the trainees' feedback would be to HU Training Services. 

• Facilitators/Trainers had sufficient time to become familiar with the training program. 

• Trainees selected for the beta-test were appropriate for the training content - they were the potential users 
of the knowledge and skills presented in the program. 

Suggestions for Improving Response Rates for Beta-Test Evaluations 

Getting organizations and organization members to respond to evaluation data collection activities has always been a 
challenge for evaluators. Perhaps it is even more difficult today, as organizations struggle with various change 
initiatives and employees appear to be doing more with less. Based on this meta-evaluation, we have learned several 
things that could help improve the response rate from post-training beta-test evaluation activities, and surveys in 
particular. Using the model proposed by Tomaskovic-Devey, et al. (1994), we have classified these into issues 
related to authority, capacity, and motive at the organization and individual levels. 

Authority 

Tomaskovic-Devey, et al. (1994) showed that issues of authority resulted in high levels of non-response, 
particularly as related to headquarters’ responsibility versus field office responsibility and policies regarding 
responding to surveys. Thus, surveys sent to a specific field unit that is part of a larger organization may receive no 
response simply because members of that office may be uncertain as to whether they or someone in headquarters are 
responsible for responding. These unit members may, in addition, be uncertain as to the policies regarding 
employees completing surveys for other organizations. Therefore, providing respondents with the following 
information is critical: Identify an appropriate organization for the beta-test, and determine what approvals are 
necessary to secure the organization’s commitment, identify a representative within the organization who has 
adequate authority to motivate and incent training participants and their managers to participate in the beta-test and 
its evaluation activities. In the present case, the salespeople handled this task but often had little contact with the 
organization’s representatives once training was scheduled. In some cases, this led to inconsistent messages to 
training program participants about the importance of their responses to the evaluation instruments. Future efforts 
should focus on obtaining cooperation from the right organizations and the appropriate contact people within those 
organizations. Provide the sponsoring organization's salespeople with a set of guidelines (or manual) that outlines 
how to select an organization for participating in beta-test training programs. This document might address: The 
sponsoring organization’s current relationship and experience with the client organization. For example, priority 
may be given to those client organizations that have already established a good working relationship with the 
sponsoring organization. It is likely that beta-test organizations will feel more committed to fully participating in the 
program’s evaluation if they've worked with the organization before, the client organization's need for training in 
the content area, the client organization's current stability (e.g., not likely to experience a major layoff or 
merger). The client organization contact’s job stability and commitment to the evaluation, a person within the client 
organization who can serve as the key contact and who might oversee the evaluation activities, and specifying the 
target training audience - the group who would most benefit from the beta-test training. 

Capacity 

Although the organization and its members may have the authority to participate and may be motivated to 
respond to the client organization's evaluation requests, the burdens of organizational life may impinge on those 
good intentions. As a result, it may be prudent for evaluators to determine in advance whether the organization and 
its members have the capacity to participate in the beta-test program and its evaluation activities. 

Develop a description of what it means to participate in a beta-test training program and the client 
organization's role in the program’s evaluation. This should include clearly delineated responsibilities of each 
person representing the sponsoring organization, the client organization, the Facilitator/Trainer, the trainees, their 
managers, and anyone else who might be responsible for the program’s delivery and evaluation. 

Ensure that the trainers have enough preparation time to deliver the training in a confident and professional 
way. Encourage trainers to fully understand the evaluation requirements and the participants’ role in the evaluation. 
Trainers could be provided with specific information about the evaluation activities that they could then share with 
the trainees. 

When using surveys to collect evaluative feedback, consider alternative distribution channels. For example, if 
the training evaluation instruments are paper-based, consider distributing them via the Internet or the organization's 
Intranet (Buchanan & Smith, 1999; Schaefer & Dillman, 1998). Or, consider offering a telephone interview to those 
who say that they are too busy to complete a survey; for those too busy to be interviewed via the telephone, provide 
a mail or electronic option (Parcel, Kaufman, & Jolly, 1991; Spaeth & O'Rourke, 1994). 
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Limit the amount of information requested on the evaluation surveys. The time to complete these may be 
viewed as excessive and could contribute to lower response rates. Guidelines for the length of an organizational 
survey vary from 30 to 150 items (see Church & Waclawski, 2001; Kraut, 1996; Paul & Bracken, 1995). In a 
recently reported series of studies. Church and Waclawski (2001) showed that survey length affected the number of 
items skipped in a survey, particularly toward the end of longer surveys. 

Motivation 

The final factor relates to the organizations’ and training participants’ motivation to participate in the beta-test 
training program and its evaluation activities. Even when the authorization and the capacity exist to respond to an 
evaluation effort, the organization or its members may not see any benefits to participating. For example, if efforts 
are not made to communicate the reasons why the organization has decided to participate in the beta-test program, 
then training participants may not realize the importance of the program or its evaluation, and thus not be motivated 
to provide timely and valid information. 

• Provide needed incentives to the organization and the organization’s representative. In this case, organizations 
were offered “free” training. It may be important for the sponsoring organization to explicitly articulate how the 
participating organizations will benefit from their involvement in the beta-test and the evaluation activities. 

• Consider the advantages and disadvantages of offering trainees and their managers a personal incentive to 
participate in the evaluation activities. The fact that employees receive free training may not be sufficient 
motivation for individuals to complete the evaluation surveys. The sponsoring organization might work with the 
client organization's representative to determine what kind of incentive would motivate trainees and their 
managers to participate in the evaluation. 

Discussion 

As long as organizations continue to pilot- and beta-test their new products and programs, they will be seeking ways 
to collect evaluative feedback. As organizations have downsized and implemented cost-saving measures, employees 
have found themselves with increased responsibilities and less time in which to accomplish their goals. If we hope to 
engage organizations in beta-testing activities, and expect them to provide useful information regarding the 
program’s strengths and weaknesses, then we must find ways to ensure that they understand their role and the 
importance of the data they are asked to provide. We hope that by employing the strategies described above, 
sponsoring organizations will experience generous response rates and high quality evaluative feedback. By 
increasing response rates, the evaluator will avoid common problems associated with non-response bias (Dooley & 
Lindner, 2003; Gall, Gall, & Borg, 2003). In addition, the higher response rates will help improve the credibility and 
value of the evaluation findings by including responses from all organizations and all organization members. Such 
evaluation findings can, in turn, be used to improve the quality of training programs provided by supplier 
organizations and training initiatives offered by customer organizations to their employees. 

This study, furthermore, demonstrates the usefulness of a meta-evaluation to examine issues arising in the 
original evaluation. Such meta-evaluations can aid HRD evaluators to improve their work and its usefulness. In 
addition, the meta-evaluation results can aid organizations in their own evaluative work and learning. 
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